{
  "nbformat": 4,
  "nbformat_minor": 2,
  "metadata": {},
  "cells": [
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "   <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/master/examples/model_experiments/model_predictions_to_project.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
        "</td>\n",
        "\n",
        "<td>\n",
        "<a href=\"https://github.com/Labelbox/labelbox-python/tree/master/examples/model_experiments/model_predictions_to_project.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Import Model Run Predictions To a Project\n",
        "Throughout the process of training your machine learning (ML) model, you may want to export your model-run predictions and import them to your new project. In this notebook, we will demonstrate the process on how to get those predictions moved over."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "!pip install -q \"labelbox[data]\""
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "import labelbox as lb\n",
        "import labelbox.types as lb_types\n",
        "from labelbox.schema.conflict_resolution_strategy import ConflictResolutionStrategy\n",
        "import uuid"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## API Key and Client\n",
        "See the developer guide for [creating an API key](https://docs.labelbox.com/reference/create-api-key)."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Add your API key\n",
        "API_KEY = \"\"\n",
        "# To get your API key go to: Workspace settings -> API -> Create API Key\n",
        "client = lb.Client(api_key=API_KEY)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Creating Model Experiment\n",
        "\n",
        "In order to interact with Model Run predictions, you must create a Model Experiment with a Model Run and then add predictions. The steps below go over this process. See [Model](https://docs.labelbox.com/reference/model) from our developer guides for more information.\n",
        "\n",
        "To create a Model Experiment you will need to create an ontology. See [Ontology](https://docs.labelbox.com/reference/ontology) for more information"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "#### Ontology"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "In this example we are making a simple ontology with a classification feature. The classification feature has two options: option 1 and option 2."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "classification_features = [\n",
        "    lb.Classification(\n",
        "        class_type=lb.Classification.Type.CHECKLIST,\n",
        "        name=\"Demo Feature\",\n",
        "        options=[\n",
        "            lb.Option(value=\"option 1\"),\n",
        "            lb.Option(value=\"option 2\")\n",
        "        ]\n",
        "    )\n",
        "]\n",
        "\n",
        "ontology_builder = lb.OntologyBuilder(\n",
        "    tools=[],\n",
        "    classifications=classification_features\n",
        ")\n",
        "\n",
        "ontology = client.create_ontology(\n",
        "  \"Demo Ontology\",\n",
        "  ontology_builder.asdict(),\n",
        "  media_type=lb.MediaType.Image\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "#### Model Experiment"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "model = client.create_model(\n",
        "  name = f\"Model Experiment Demo {str(uuid.uuid4())}\",\n",
        "  ontology_id=ontology.uid\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Creating a Model Run from Model Experiment\n",
        "\n",
        "On this step we will need to create a dataset to attach data rows to our model run. See [Dataset](https://docs.labelbox.com/reference/dataset) for more information."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "#### Dataset and Data Rows"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# send a sample image as data row for a dataset\n",
        "global_key = \"2560px-Kitano_Street_Kobe01s5s4110\"\n",
        "\n",
        "test_img_url = {\n",
        "    \"row_data\":\n",
        "        \"https://storage.googleapis.com/labelbox-datasets/image_sample_data/2560px-Kitano_Street_Kobe01s5s4110.jpeg\",\n",
        "    \"global_key\":\n",
        "        global_key\n",
        "}\n",
        "\n",
        "dataset = client.create_dataset(name=\"foundry-demo-dataset\")\n",
        "task = dataset.create_data_rows([test_img_url])\n",
        "task.wait_till_done()\n",
        "\n",
        "print(f\"Errors: {task.errors}\")\n",
        "print(f\"Failed data rows: {task.failed_data_rows}\")\n",
        "\n",
        "if task.errors:\n",
        "    for error in task.errors:\n",
        "        if 'Duplicate global key' in error['message'] and dataset.row_count == 0:\n",
        "            # If the global key already  exists in the workspace the dataset will be created empty, so we can delete it.\n",
        "            print(f\"Deleting empty dataset: {dataset}\")\n",
        "            dataset.delete()"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "#### Create Model Run and Attach Data Rows"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "model_run_name = f\"Model Run Demo {str(uuid.uuid4())}\"\n",
        "\n",
        "model_run = model.create_model_run(name=model_run_name)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "model_run.upsert_data_rows(global_keys=[global_key])"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "#### Add Predictions\n",
        "In the below code snippet we are adding a sample predictions and attaching them to our data row inside our model run."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "checklist_prediction = lb_types.ClassificationAnnotation(\n",
        "    name=\"Demo Feature\",\n",
        "    value=lb_types.Checklist(answer = [lb_types.ClassificationAnswer(name = \"option 1\", confidence=0.5)])\n",
        ")\n",
        "\n",
        "# Create prediction label\n",
        "label_prediction = [\n",
        "    lb_types.Label(\n",
        "        data = lb_types.ImageData(global_key=global_key),\n",
        "        annotations = [checklist_prediction]\n",
        "    )\n",
        "]\n",
        "\n",
        "# Upload the prediction label to the Model Run\n",
        "upload_job_prediction = model_run.add_predictions(\n",
        "    name=\"prediction_upload_job\"+str(uuid.uuid4()),\n",
        "    predictions=label_prediction)\n",
        "\n",
        "# Errors will appear for prediction uploads that failed.\n",
        "print(\"Errors:\",  upload_job_prediction.errors)\n",
        "print(\"Status of uploads: \", upload_job_prediction.statuses)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "### Setup Project and Add Predictions\n",
        "In the steps below we will be creating our target project and setting up the project with the ontology we used with our model run. See [Project](https://docs.labelbox.com/reference/dataset) for more information."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "#### Project "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Create a new project\n",
        "project = client.create_project(\n",
        "    name=\"Model Run Import Demo Project\",\n",
        "    media_type=lb.MediaType.Image\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "# Setup Ontology\n",
        "project.setup_editor(ontology)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "#### Ontology Mapping\n",
        "To send prediction to your annotate project you will need to provide a ontology mapping python dictionary item. This matches ontology feature id to another. You would use this if your ontology was different from your model run to your project. In our case, since we are using the same ontology, you would just need to map the same feature id to each other."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Get ontology dictionary to obtain featureSchemaIds\n",
        "ontology_normalized = ontology.normalized\n",
        "\n",
        "PREDICTIONS_ONTOLOGY_MAPPING = {\n",
        "    ontology_normalized[\"classifications\"][0][\"featureSchemaId\"]:ontology_normalized[\"classifications\"][0][\"featureSchemaId\"], # Classification featureSchemaID\n",
        "    ontology_normalized[\"classifications\"][0][\"options\"][0][\"featureSchemaId\"]:ontology_normalized[\"classifications\"][0][\"options\"][0][\"featureSchemaId\"], # Different Classification Answer featureSchemaIDs\n",
        "    ontology_normalized[\"classifications\"][0][\"options\"][1][\"featureSchemaId\"]:ontology_normalized[\"classifications\"][0][\"options\"][1][\"featureSchemaId\"]\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "#### Send Model Predictions from Model Run to Annotate\n",
        "To send our predictions to our project we will be using the `send_to_annotate_from_model` method from our project. See [Foundry apps](https://docs.labelbox.com/reference/foundry-2#send-foundry-generated-annotations-from-catalog-to-annotate) for more information.\n",
        "##### Parameters\n",
        "\n",
        "When you send predicted data rows to annotate from a model run, you may choose to include or exclude certain parameters, at a minimum a predictions_ontology_mapping will need to be provided:\n",
        "\n",
        "* `predictions_ontology_mapping`\n",
        "    - A dictionary containing the mapping of the model's ontology feature schema ids to the project's ontology feature schema ids\n",
        "* `exclude_data_rows_in_project`\n",
        "    - Excludes data rows that are already in the project. \n",
        "* `override_existing_annotations_rule` \n",
        "    - The strategy defining how to handle conflicts in classifications between the data rows that already exist in the project and incoming predictions from the source model run or annotations from the source project. \n",
        "        * Defaults to ConflictResolutionStrategy.KeepExisting\n",
        "        * Options include:\n",
        "            * ConflictResolutionStrategy.KeepExisting\n",
        "            * ConflictResolutionStrategy.OverrideWithPredictions\n",
        "            * ConflictResolutionStrategy.OverrideWithAnnotations\n",
        "* `param batch_priority`\n",
        "    - The priority of the batch.\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "#### Import Predictions as pre-labels"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "send_to_annotations_params = {\n",
        "    \"predictions_ontology_mapping\": PREDICTIONS_ONTOLOGY_MAPPING,\n",
        "    \"exclude_data_rows_in_project\": False,\n",
        "    \"override_existing_annotations_rule\": ConflictResolutionStrategy.OverrideWithPredictions,\n",
        "    \"batch_priority\": 5,\n",
        "}\n",
        "\n",
        "# Send the predictions as pre-labels\n",
        "queue_id = [queue.uid for queue in project.task_queues() if queue.queue_type == \"INITIAL_LABELING_QUEUE\" ][0]\n",
        "\n",
        "task = model_run.send_to_annotate_from_model(\n",
        "    destination_project_id=project.uid,\n",
        "    task_queue_id=queue_id, # ID of workflow task, set ID to None if you want to convert pre-labels to ground truths .\n",
        "    batch_name=\"Prediction Import Demo Batch\",\n",
        "    data_rows=lb.GlobalKeys(\n",
        "        [global_key] # Provide a list of global keys from foundry app task\n",
        "    ),\n",
        "    params=send_to_annotations_params\n",
        "    )\n",
        "\n",
        "task.wait_till_done()\n",
        "\n",
        "print(f\"Errors: {task.errors}\")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Cleanup"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "#project.delete()\n",
        "#dataset.delete()\n",
        "#model_run.delete()"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    }
  ]
}